A file transfer program is one in which a file is transferred between an application running on a computer and an I/O device in a read or write operation. The file being transferred comprises records grouped into blocks, wherein the blocking of the records is not important to the logical content of the file. The records may be blocked differently depending on where the file is stored. In a source file, the records may be blocked in one way, they may be blocked in another way while being transferred from the source to a destination, and they may be blocked in still another way when they are stored in the destination file.
Five specifications for file transfer programs which are difficult to support at the same time are:
1. A truly generic file transfer program allows users to specify their own I/O routines to read and write files. In this way, the file transfer program can read from and write to any I/O device for which the user has written an I/O routine. PA1 2. When large amounts of data are being transferred, the file transfer program should allow for recovery in case the transfer is stopped for some reason. For example, a transfer might be stopped because a communication mechanism like TCP/IP has crashed, or because a machine must be recycled. When this occurs, it is often desirable to resume the transfer at a later time from the point at which it was stopped. PA1 3. The file transfer program should support mechanisms for monitoring the transfer as it proceeds such as indicators of how far along the transfer is and at what rate it is transferring the file. PA1 4. The file transfer program should support mechanisms for changing the contents of files as they are transferred such as stride, beginning and ending record number, and user supplied conversion routines. PA1 5. The file transfer program should support transfers for file types which do not allow for recovery, such as named pipes useable with the IBM AIX operating system and batch pipes useable with the IBM MVS operating system. Files of these types do not allow a program to index into the middle of a file to start reading or writing from that point.
Therefore, there is a requirement for an interface to a generic I/O routine that can handle many types of I/O.
These five specifications work together to create many difficulties. For instance, at recovery time, parameters, referred to herein as status, used for monitoring the transfer must be set to the correct values so monitoring of status can be resumed. Also, the file changing algorithms (hereinafter referred to as the conversion routine) need to be reset to the correct value from the point at which the transfer must be resumed. Also, the values that the I/O routine must return for recovery may not correspond to the most recent values of the monitoring and file changing information. The I/O routine may need to recover to a point in the transfer well before the one to which the transfer had proceeded when it was stopped. Consider, for example, the file of FIG. 1 which contains blocks 10, 11, and 12. Block 10 includes records 1-6, block 11 includes records 7-12, and block 12 includes records 13-17. A file transfer program will pass the records of this file to an I/O routine, one at a time, for writing to some storage medium. Each block will be placed into memory before it is written to the storage medium. When record 1 is passed to the I/O routine, the routine will place the record in a block of memory, but will not yet write the record to the storage medium. Thus, at this point the I/O routine cannot return recovery information for record 1 to the file transfer program because record 1 has not yet been written to the storage medium. Likewise, records 2 through 5 are passed by the file transfer program, and are also stored in the block of memory, but are still not moved to the storage medium. Thus, for records 1 through 5, no recovery information can be returned to the file transfer program. Only when record 6 has been passed to the I/O routine can the routine store records 1 through 6 in the storage medium. It is then that recovery information can be sent to the file transfer program. The recovery information that is available at that time is the location on the storage medium at the beginning of the block, which is the location of the beginning of record 1 on the storage medium. An example of a file access method that provides this kind of information is the BSAM macros on the MVS operating system by IBM. The BSAM NOTE macro returns the position of the beginning of the block most recently written to disk or tape.
In a similar way, records 7 through 12 are written to the storage medium when record 12 is passed to the I/O routine. The file access method will return the position on the storage medium of record 7, which the I/O routine then passes back to the file transfer program.
Consider the situation where records 13 through 15 are passed to the I/O routine, and are blocked in memory. If the file transfer stopped at this point for any reason, recovery must start with record 7 rather than record 16 because records 13 through 15 were never written to the storage medium, and thus cannot be recovered. Records 8 through 12 cannot be recovered because present access methods do not allow recovery to the middle of a block. In addition, though the file transfer's status information and file changing algorithms have progressed to record 15, they must be reset to record 7 so that the transfer can be resumed from record 7.
Some files do not allow a program to index into the middle of a file to start reading or writing. Recovery methods which involve indexing into the middle of both the source file and the destination file thus are not suitable for these types of files.